Evaluation at Large Scale FP 7 – 238975 D 13 . 1 Evaluation design and collection of test data for semantic

for dissemination) This deliverable presents a systematic procedure for the evaluation of semantic search tools within the SEALS framework. A review of current search tool technologies and previous evaluation efforts is presented which informs the approach adopted later in the document. The document describes how the SEALS evaluation methodology will be applied by describing the criteria by which the tools will be assessed and identifying the specific metrics and interpretations that will be employed. In addition, we describe a number of potential data sets and identify those which are suitable for our evaluation requirements. A set of reference benchmark tests are described for assessing the strengths and weaknesses of the available tools and to compare them. As such, these tests focus on the performance of fundamental aspects of the tool in a strictly controlled environment / scenario rather than their ability to solve open-ended, real-life problems. These fundamental aspects are the formal evaluation criterion by which we will benchmark each tool: • Query expressiveness • Usability (effectiveness, efficiency, satisfaction) • Scalability • Interoperability • Suitability for the Semantic Web • Quality of documentation In common with the evaluation procedure employed by many of the other tool type evaluation teams within SEALS, an automated evaluation approach will be adopted. However, in order to draw useful conclusions regarding the usability of a tool, it is essential that real users are asked to complete tasks using the tool. Therefore, for the purposes of evaluating semantic search tools, we adopt a two phase approach: an automated phase and a user-in-the-loop phase. The user-in-the-loop phase involves a series of experiments involving human subjects (conducted by the tool provider) who are given a number of tasks (questions) to solve and a particular tool and ontology with which to do it. Since the evaluation consists of two distinct phases each addressing different criteria, each phase uses a different data set to ensure optimal applicability to each task: • Mooney (user-in-the-loop) • EvoOnt (automated) The Mooney data set is well established in semantic search tool evaluations and has an appropriate set of questions for which the groundtruth is known. It is an ideal data set for investigating the usability of a tool interface. However, its limited size means it is less well suited to criteria such as scalability. Therefore, the EvoOnt dataset which can be significantly larger than the Mooney data set is used. User-in-the-loop user evaluations necessitate the provision of additional software above and beyond that provided by the SEALS consortium as a whole. Additional software is required to both run the experimental workflows and also obtain test data and return results data to the various SEALS repositories.